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With an increase in utilization and hours of payload operations being executed onboard 
the International Space Station (ISS), upgrading the NASA Marshall Space Flight Center 
(MSFC) Huntsville Operations Support Center (HOSC) ISS Payload Control Area (PCA) 
was essential to gaining efficiencies and assurance of current and future payload health and 
science return. PCA houses the Payload Operations Integration Center (POIC) responsible 
for the execution of all NASA payloads onboard the ISS. POIC Flight Controllers are 
responsible for the operation of voice, stowage, command, telemetry, video, power, thermal, 
and environmental control in support of ISS science experiments. The methodologies and 
execution of the PCA refurbishment were planned and performed within a four-month 
period in order to assure uninterrupted operation of ISS payloads and minimal impacts to 
payload operations teams. To vacate the PCA, three additional HOSC control rooms were 
reconfigured to handle ISS real-time operations, Backup Control Center (BCC) to Mission 
Control in Houston, simulations, and testing functions. This involved coordination and 
cooperation from teams of ISS operations controllers, multiple engineering and design 
disciplines, management, and construction companies performing an array of activities 
simultaneously and in sync delivering a final product with no issues that impacted the 
schedule. For each console operator discipline, studies of Information Technology (IT) tools 
and equipment layouts, ergonomics, and lines of sight were performed. Infusing some of the 
latest IT into the project was an essential goal in ensuring future growth and success of the 
ISS payload science returns. Engineering evaluations led to a state of the art Video Wall 
implementation and more efficient ethernet cabling distribution providing the latest 
products and the best solution for the POIC. These engineering innovations led to cost 
savings for the project. Constraints involved in the management of the project included 
executing over 450 crew-hours of ISS real-time payload operations including a major 
onboard communications upgrade, SpaceX un-berth, a Soyuz launch, roll-out of ISS live 
video and interviews from the POIC, annual BCC certification and hurricane season, and 
ISS simulations and testing. Continuous ISS payload operations were possible during the 
PCA facility modifications with the reconfiguration of four control rooms and standup of 
two temporary control areas. Another major restriction to the project was an ongoing 
facility upgrade that included a NASA Headquarters mandated replacement of all electrical 
and mechanical systems and replacement of an external generator. These upgrades required 
a facility power outage during the PCA upgrades. The project also encompassed console 
layout designs and ordering, amenities selections and ordering, excessing of old equipment, 
moves, disposal of old IT equipment, camera installations, facility tour re-schedules, and 
contract justifications. These were just some of the tasks needed for a successful project. 
This paper describes the logistics and lessons learned in upgrading a control center 
capability in the middle of complex real-time operations. Combining the efficiencies of 
controller interaction and new technology infusion were prime drivers for this upgrade to 
handle the increased utilization of science research on ISS. The success of this project could 
not jeopardize the current operations while these facility upgrades occurred. 
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I. Introduction 

T HE National Aeronautics and Space Administration (NASA) International Space Station (ISS) Payload 
Operations Integration Center (POIC) began operations on the 26 th of February 2001 during ISS Increment 5A- 
1. Since then, the POIC has operated 24 hours a day, 7 days a week, 356 days a year. The POIC is located at 
Marshall Space Flight Center (MSFC) in Huntsville, Alabama in the Huntsville Operations Support Center (HOSC). 
POIC is the control center executing command and control of ISS science research. After 12 years of ISS payload 
operations, an upgrade of the center was needed. 

In 2009, research and inquiries into 
specific information technology (IT) 
upgrades and the latest control room 
technologies had been investigated, but with 
no funding available, only a “wish list” had 
been created. The increase in ISS utilization 
and hours of payload science operations 
being executed onboard the ISS drove a need 
to evaluate and expand the possibilities of 
upgrading the ISS Payload Control Area 1 
(PCA-1). This was becoming more essential 
to gaining efficiencies and assurance of 
current and future payload health and science 
return. The responsibility of the ISS payload 
flight controllers was increasing and things 
fell into place for the possibility of the re- 
engineering of the PCA-1 to become a 
reality. With the complete redesign and refurbishment, there would be many challenges, two of the biggest, ensuring 
uninterrupted 24x7x365 ISS payload on-orbit operations and certification and readiness for the 2013 Backup 
Control Center (BCC) to Mission Control Center Houston (MCC-H), located at Johnson Space Center (JSC). 

II. Project Management 

Project Management (PM) for this upgrade began 11 months prior to the unveiling of the new ISS payload 
operations control area, but providing NASA mission control and science centers and research and development for 
those control room technologies had been happening at the mission operations ground systems HOSC facility since 
the NASA Spacelab Program which initiated our first command and control operation. In addition, throughout the 
years, meetings and interchanges had been held with ISS International Partner (IP) mission control centers including 
ISS Mission Control Center Moscow located in Korolov, Russia, which contains three ISS control rooms, the Japan 
Aerospace Exploration Agency (JAXA) Space Station Operations Facility at JAXA’s Tsukuba Space Centre, the 
European Space Agency (ESA) German Aerospace Columbus Control Centre at Oberpfaffenhofen, Germany, and 
the French Space Agency Centre National d’etudes Spatiales Toulouse Space Centre Automated Transfer Vehicle 
(ATV) Control Centre. The HOSC had experience in supporting and assisting a multitude of ISS science operations 
centers both within the United States (US) and internationally with their systems and experiment operations control 
configurations. In addition, other US government facilities, NASA centers, and commercial control centers were 
visited and provided their control center development experiences. Among those were SpaceX flight control and US 
Army control rooms. JSC was undergoing the development of new MCC-H JSC flight control rooms. The interface 
and interchanges between MSFC and JSC proved to be critical in the delivery of the PCA-1. JSC provided lessons 
learned and technical support to MSFC, which expedited the execution of the PCA-1 delivery. With these levels of 
exposure and experience with space mission control centers, MSFC Mission Operations Laboratory (MOL) was 
confident the challenges could be met. The first steps in the upgrade were to receive MSFC center management and 
ISS Program authority, secure funding sources, and identify risks and constraints. 

A. Risks and Constraints 

The biggest challenge that had to be met was continuous uninterrupted ISS payload operations. The Integrated 
Communications Units (ICUs) were installed onboard the ISS during the PCA-1 upgrade. There was regular vehicle 
traffic during this timeframe including Dragon and Soyuz departures, two different Soyuz arrivals, a Progress arrival 
and subsequent departure, and an ATV launch. These are just some of the critical operations events that occurred 
while flight controllers were out of their normal operating environment. 
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Figure 1. Photograph of PCA-1 in 2002. 



Another major support activity for the HOSC is the ISS BCC. The BCC to NASA’s MCC-H, located in 
Huntsville, is a top priority Marshall Space Flight Center service. BCC training and certification occur annually 
during the month of May, prior to the start of the Hurricane Season, which runs from June 1 to November 30. This 
was a major requirement for project completion by June 1, 2013 as the PCA-1 payload flight controllers were moved 
into another control area during the upgrades. 

In addition, a separate challenge existed with facility upgrades happening at the HOSC. The MSFC HOSC 
facility was built in 1959. Several mission control rooms and areas for a variety of NASA space missions are 
supported within the HOSC. The HOSC facility was undergoing an 18 month construction project to raise the 
overall facility availability rating. Due to the age of the facility, the HOSC requires an annual facility power outage 
for maintenance, which assures mission availability requirements are met and no unplanned outages occur. The 
upgrade replaced all mechanical and electrical systems within the building, providing a dual power path which 
eliminated the need for annual power outages. To execute this construction project, two complete power downs were 
required during the PCA-1 upgrade. The HOSC developed a smaller scale operations control area, HOSC Power 
Outage Contingency (HPOC), in an alternate building so ISS operations could continue. 

Control room moves would be required to keep ISS operations, simulations, and BCC operating. Minimal 
disruption was a necessity. The HOSC Payload Control Area 2 (PCA-2) simulation and training control area mirrors 
the PCA-1. Moving PCA-1 payload flight controllers to PCA-2 seemed the least disruptive to the ISS team, but 
without PCA-2 for simulations and training, there would be impacts to many external parties. The HOSC had 
recently closed out satellite payload mission operations for the Fast, Affordable, Science, and Technology Satellite 
(FASTSAT) mission that launched in 2010 and successfully ran for 18 months. FASTSAT operations were 
conducted in the HOSC Configurable Control Room (CCR). HOSC training and simulation teams would 
temporarily reduce requirements so as to fit into the smaller CCR. This reduction in training and simulation 
capability increased the overall risk, with an aggressive schedule. PCA-2 also hosted the MCC-H BCC. So, if this 
aggressive schedule were not met, the work would fall into Hurricane Season, potentially impacting our BCC 
support to JSC. To alleviate this risk, JSC and MSFC BCC teams worked together to evaluate the HOSC Shuttle 
Engineering Support Center (SESC) for upgrades to move the BCC capability into the SESC for the 2013 Hurricane 
Season. The SESC was the operations control room that provided pre-launch and launch support to Kennedy Space 
Center launch control during the Space Transportation System, or Shuttle, Program. This control area will also 
support the NASA Space Launch System (SLS) Program. Impacts and constraints were analyzed and evaluated in 
the development of an implementation plan, with internal and external constraints defined as listed in Table 1. 


Table 1. Internal and External Constraints 

External Constraints 

1 . BCC Training, Certification, Hurricane Season 

2. MCC-H Situational Awareness Video Feed of PCA 

3. Public Affairs Office ISS Live Video Commitment 

4. Space and Rocket Center tours 


Internal Constraints 

5. Real Time Operations, Control Room downtime 

6. Simulations 

7. Stress due to moves of people and equipment 

8. Aggressive Schedule 


Three options with schedules were developed involving available HOSC mission control areas, as listed in Table 2 

Table 2. Three options for people and equipment moves to support PCA-1 re-engineering. 


Option 1 

• Requires 5 equipment moves and 2 
console personnel moves 

• Alleviates constraints 1,2, 3, 4, 7 

• Does not alleviate constraints 5,6,8 


Option 2 

• Requires 3 equipment moves and 2- 
4 console personnel moves 

• Alleviates constraints 1,2, 3, 4, 7, 8 

• Does not alleviate constraints 5,6 

• PCA-2 remains stable 


Option 3 - Forward Plan 

• Requires 4 equipment moves and 2 
console personnel moves 

• Alleviates constraints 1,2, 3, 4, 5, 6, 7, 8 

• PCA-2 remains stable 

• BCC is stable 


B. Sprint Productivity Development Technique 

With a multitude of facets, organizations, and challenges, the PCA-1 upgrade team chose to follow a 
development technique that is typically used in software development projects. The Sprint development method is a 
flexible, iterative, and incremental approach to product development where a team works very closely together 
reaching a common goal. The Sprint method allows for effective and efficient face-to-face communications and is 
directed by a lead, who suggests tasks, and tracks progress assuring impacts are assessed and addressed quickly. 
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The first PCA-1 Sprint was 5 days long involving team members from each flight controller position, 
engineering disciplines, construction team, and potential vendors for products. All stakeholders were considered 
members of the team. The earlier a facet of the project was defined, the sooner a team representative was brought in 
as a member of the Sprint team. The team quickly laid out a schedule addressing dependencies from each 
stakeholder. After the first 5 day Sprint, execution of the project began and Sprints went to a bi-weekly, weekly, and 
as needed basis, analyzing progress and making required adjustments. 

A key to recognize is that during a Sprint project, customers change their requirements causing unpredicted 
challenges. Accepting that a problem may not be fully understood, defined, or solved at any particular point in the 
schedule requires the team’s flexibility to focus on responding and delivering quickly. 

This focused development and face-to-face Sprint method expedited the planning, design, and development, 
decreasing the risk of not meeting schedule. 

C. Budget 

At the end of the fiscal year, ISS program funds were provided to MSFC for the execution of the upgrade to 
occur during the following fiscal year. The ISS Program provided eighty percent of the funding and the MSFC 
Engineering Directorate provided the remaining investment. 

For budget management purposes, three areas of funding for project management were defined; 1 - Consoles, to 
provide for Consoles, wall bookcases, printer tables, wedge tables, console fitted bookcases, and lockers; 2 - 
Hardware and Software, to provide for the Video Wall implementation, cables, controllers, and processor software; 
and 3 - Facilities, to provide for the reconstruction of the duct and sprinkler systems above the video wall, room 
amenities such as wall designs, all the physical moves among control rooms, paint, and carpet. 


D. Schedule 

There was a great deal of work to be done to reengineer PCA-1 in 2013. Listed below is a short summary of 
important dates, followed by detailed explanation of the events that occurred. 

• February 1 PCA-1 upgrade kickoff 

• March 27 Flight Cadre moves from PCA-1 to PCA-2 

• June 7 PCA-1 upgrade complete 

• June 1 1 Flight Cadre moves back to PCA-1 

• June 19 PCA-1 Media Event and Unveiling 


During February and March, the team spent time planning for the PCA-1 redesign so that operations would run 
smoothly and efficiently. Site visits of other control rooms were performed. The team made room layout, console, 
and video wall decisions. The SESC was reconfigured to support BCC. The FASTSAT equipment was cleared out 
of CCR so that simulations could be stood up in that location as expeditiously as possible once hardware from PCA- 
1 was available. Meetings and checkouts were held in PCA-2 to enusure that systems were ready for transition to 
flight operations. The cadre transition from PCA-1 to PCA-2 occurred while the ISS crew was asleep on the evening 
of March 27. That particular date was chosen so as not to impact the SpaceX-2 unberth the day before or the Soyuz 
launch the day after. After the cadre vacated PCA-1, the first order of business was to hang a time-lapse camera to 
record the progress. Then the computer equipment and furniture was moved to the CCR to support simulations. 
Simulations resumed on April 15 in the CCR. US Space and Rocket Center tours were discontinued from the time 
the cadre moved out of PCA-1 until after the unveiling. 

The console and video wall orders were finalized on April 1 and April 2, respectively. The month of April was 
mainly devoted to facilty upgrades and modifications. The ducts and sprinklers along the north wall of PCA-1 were 
moved to accomate the video wall, which needed to be as high up the wall as possible for operator line of sight. The 
weekend of April 12-14, the HOSC facility was shut down for a complete power outage so that redundant power 
systems could be installed. During this time, the cadre moved to the HPOC. Onboard the ISS, the first two weeks of 
April were ICU installation weeks, so the cadre was exceptionally busy. The redesign team worked with the Office 
of Strategic Analsysis and Communications to come up with a mural for the west wall that would make the POIC 
recognizable during ISS Live video. 

During the month of May, the new PCA-1 started to take shape. The room was painted. The video wall hardware 
was delivered on May 13 and was installed by May 17. Carpet was installed on May 17. The consoles were 
delivered on May 20 and installed by May 23. In other rooms throughout the HOSC, BCC certification was 
completed on May 23, in preaparation for Hurricane season and the FASTSAT team had a pass with the satellite to 
confirm its end of life. 
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By May 28, the network configuration in PCA-1 was ready for equipment, so simulations in the CCR stood 
down so that equipment could be used to build up PCA-1. PCA-1 was checked out for operations on June 10 and the 
cadre transitioned to the new room the next day after a Progress undock and during crew sleep. PCA-2 was go to 
resume simulations on June 12. 

On June 19, the ISS Program Manager, Mike Suffredini, attended the unveiling of the new control room along 
with the MSFC Deputy Center Director, Engineering Director, ISS Payloads Manager, the MSFC Mission 
Operations Director, the Public Affairs Office, news stations, and newspapers. 


III. Design Approach and Considerations 

The POIC is responsible for the execution of all NASA payloads onboard the ISS. POIC Flight Controllers are 
responsible for the operation of voice, stowage, command, telemetry, video, power, thermal, and environmental 
control in support of ISS science experiments. Through the years, ISS payload ground systems and flight controller 
computer hardware executing these functions have been updated and refreshed staying on top of the IT needs of 
flight controllers. Operator consoles, display technologies, operator’s line of sight between positions, operator’s 
visibility of large screen wall displays, console equipment, corridor widths, traffic flow, and travel routes for 
collaborative operations among operators, viewing area, storage, and room amenities were areas that needed 
attention and upgrading. The control room provides operators a safe and functional environment helping operators to 
run their operations more efficiently. The room must be designed accordingly. With 12 years of ISS Payload 
operations under our belts, there was a plethora of lessons learned, ideas, and improvements to be considered. 

Research was conducted and leveraged through investigation of other control centers design and technologies. 
The Re-engineering team facilitated extensive discussions and information gathering through Technical Interchange 
Meetings, tours, and online research with other government and commercial operating control rooms and vendors. 
Operator consoles, display technologies, line of sight between flight control positions, operator’s visibility of large 
screen wall displays, console equipment, corridor widths, traffic flow, travel routes for collaborative discussions 
among operators, viewing area, storage, and room amenities all had to be considered. Conceptual drawings were 
created with a 3D drafting tool for studies on corridor widths, traffic flow, collaborative operations, and storage. 


A. Existing PCA-1 Arrangement 

The existing PCA-1 
consoles were 12 years old 
and not designed to 
expand to meet ISS 
payload flight controller 
increasing operations 
concepts. Console tops 
were overcrowded. Two 
projectors were being used 
to display information that 
was not being utilized by 
the console operators 
because the quality of the 
projectors and 

arrangement of the 
consoles did not allow 
ease of use. Video wall 
technologies had been 
investigated and it was a 
big desire to have a video 
wall installed as part of the 
upgrade. Larger screen 
monitors had been 
mounted around the walls 
of the room in an ad hoc 

manner, so it only made sense to seriously consider a video wall implementation. Other constraints that had to be 
considered were two structural columns in the room that were not removable, and whether to remove the flags 
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Figure 2. Drawing of PCA-1 before redesign. 
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mounted on the ceiling that represented all ISS partners. Discussions on rotating the room 90 degrees were 
underway. Lastly, there was the PCA-1 Viewing Room to consider. The Viewing Room is adjacent to PCA-1 on the 
east wall, separated by large windows. This room is used by internal and external tour groups, including the US 
Space and Rocket Center, which brings paying tourists for a behind the scenes look at MSFC. Tour groups would 
only get a really great view of the console operator’s back who was stationed on the back row of the room and not 
much else, except the tops of other flight controllers heads. 

B. PCA-1 Re-Design Drafting 

A 3D Computer Aided Drawing (CAD) tool was used to analyze orientation of the room, possible console 
widths and depths, storage, and amenities. With all the options to consider, the team realized that turning the room 
90 degrees clockwise, facing the north wall, would give more area for a video wall, meaning more information could 
be displayed and shared among the cadre. It also meant that the operators in the back row were much closer than 
they would be if the video wall were on the west wall. Lastly, turning the layout 90 degrees provided a better view 
from the viewing room because the consoles were arranged perpendicular to the glass, giving a tour group a view 
down the entire length of each row. 

The CAD tool was also used to analyze lines of sight between operators and operators to the video wall. 
Operator blueprints for 
how operators interface 
with systems, other flight 
controllers, analysis of 
data sources, and the 
processes required 

executing operations also 
fed the size and number of 
displays the video wall 
needed. The conclusion of 
all the analysis was that an 
information rich display 
solution that was easy to 
use with intelligent 
controls would increase 
operator efficiency and 
reduce response time. 

Construction would be Figure 3. Initial 3D conceptual rendering of PCA-1. 
required to modify the 

facility duct and sprinkler systems, raising them to accommodate the video wall. 

The selection of the company providing operator consoles was selected partly due to their extensive experience 
in large-scale and complex projects with a uniquely experienced team of managers, engineers, installers, and 
suppliers. The vendor also specialized in operator ergonomics and suggested hardware configurations. 

C. Impacts to Operations 

There were several critical ISS events known to the project management team going into the redesign effort. 
These critical events had to be planned around so that work would not interfere with the onboard operations. It was a 
big challenge and critical to the success of the PCA-1 re-engineering project. Not only was there the usual work of 
managing and operating the science experiments on the ISS (no small feat on any given day), there was also a steady 
stream of vehicle traffic and the biggest upgrade to the high rate data system the ISS had ever seen. 

The POIC is responsible for planning, integration, and operation of the high rate data system on the ISS. This 
Ku-band system is how the vast amounts of scientific data and video get from the ISS to the ground and eventually 
to scientists all over the world. The Data Management Coordinator (DMC) is the POIC console position responsible 
for the Ku-band data. The onboard communication system upgrade necessitated the enhancement of the DMC and 
Alternate DMC consoles. Transition dates were chosen around the critical events and with enough lead-time so that 
the DMC team was comfortable with the new configuration. It was critical to the success of the upgrade that the 
DMC team was as comfortable and had every tool available to them in PCA-2 as they were in PCA-1. Additionally, 
during the 2-week upgrade process, the POIC staffed 2 DMCs, which meant that there needed to be 2 fully 
functional consoles. 
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There were several ISS research operations that occurred for the first time during the PCA-1 redesign, while the 
POIC cadre was relocated to PCA-2. Robonaut, a humanoid robot experiment, was first operated via crew control 
using goggles and special gloves that sense hand movement. In essence, Robonaut mimicked the movements of the 
crewmember. Prior to this, Robonaut was controlled via ground-based commanding. This successful test paves the 
way for ground control whereby an operator uses the goggles and gloves to control Robonaut, instead of issuing 
commands at a computer. The PCA-1 redesign timeframe also saw the first simultaneous enablement of two 
scientists on the ground talking to two different ISS crewmembers while they performed different experiments. One 
crewmember was performing a flame extinguishing experiment aimed as studying how different spacecraft materials 
bum, while the other was performing a fluid physics experiment aimed at developing better fuel tanks and 
propulsion systems in space. A next generation carbon dioxide scmbber was installed and began operations, the first 
software defined radio transmitter completed checkout and began operations, and the first in a series of plant growth 
experiments began, studying how gravity and light responses influence each other. 

In addition to new science experiments, the ISS also had regular vehicle traffic during this timeframe, including 
SpaceX’s Dragon, which departed carrying home a multitude of science samples and old hardware for 
refurbishment. Six new ISS crew members arrived on 2 Soyuz flights and 3 different crewmembers departed after 
their 6-month stay. An unmanned Progress vehicle launched and docked, bringing supplies and equipment, then 
undocked 7 weeks later carrying trash. A European ATV launched during the PCA-1 refurbishment and docked after 
cadre transition back to the new PCA-1 . 

In addition to new research investigations and vehicle traffic, the ISS had other notable events including the 
deployment of high definition camcorders, which were made possible by the upgraded communications unit and an 
unplanned extravehicular activity (spacewalk) to investigate an ammonia leak in the external coolant system. The 
ability to support all of these events and firsts demonstrate the seamless transition of the cadre from PCA-1 to PCA- 
2. In effect, the cadre was out of their normal operating environment, but the planning and project management that 
went into the redesign effort gave the cadre all the tools and confidence they require of their systems. 


D. Impacts to Simulations and Training 

The HOSC generally supports two ISS simulations each week. The sims are usually performed in PCA-2. If the 
real-time cadre moved into PCA-2 during PCA-1 refurbishment, the simulations would have to find an alteranate 
location. Simulations are critical to the cadre training process for both new and experienced team members. The 
onboard communications upgrades significantly changed the job of the DMC, meaning that the entire team had to 
re-certify to operate the new equipment. It was critical that any impact to simulations be minimized so the DMC 
team could complete their required training and support real-time flight operations. 


12 13 14 15 16 17 18 19 20 21 22 23 24 25 


E. Impacts to Backup Control Center 

Since 2008, the HOSC has served as 
BCC to Mission Control in Houston. In 
that year, BCC was activated for Hurricane 
Ike, and from MSFC, Russian Progress 
3 OP was docked to the ISS during 
Expedition 17. Providing this service is a 
top priority of MSFC and the re- 
engineering of PCA-1 could not affect the 
HOSC’s ability to host BCC. Additionally, 
as the POIC cadre needed improvements to 
systems to support the onboard 
communications upgrades, so too did BCC. 

The safest option for BCC stability meant 
moving BCC for the 2013 Hurricane 
Season to the SESC and incorporating the 
necessary communications upgrades in that 
location. This meant that if PCA-1 re- 
engineering work continued past the 
annual BCC Certification run by JSC in 
May, BCC was unaffected, since the POIC 
cadre cannot operate out of the BCC room 
during certification. Effectively, the move of BCC to the SESC relieved the schedule pressure to complete the PCA- 



PCA-2 / BCC 

Figure 4. PCA-2 configuration for BCC. 
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1 re-engineering effort by June 1. The HOSC team collaborated with JSC to coordinate the installation of the new 
hardware and the network reconfigurations necessary for moving BCC to SESC. 

In 2013, JSC was engineering the delivery of a new Mission Critical Environment (MCE) that consolidated 
MCC-H mission critical systems which included the BCC system. New BCC MCE hardware had been designed, 
engineered, and installed at the HOSC. With the delivery schedule of the MCC-H BCE shifting, BCC training and 
certification for 2013 Hurricane system was required to be completed for both the BCC operational system and the 
BCC MCE re-engineering deliveries. 



Figure 5. SESC configuration for BCC. 


F. Facility Considerations 

During the PCA-1 upgrade, a facility upgrade was underway to replace all mechanical and electrical systems 
within the HOSC. This was a construction project providing a dual power path to eliminate the need for annual 
power outages. To execute this construction project, two complete power downs of the building were required. 
Although very necessary, the work was, at times, very intrusive and required substantial coordination and detail to 
schedules across various NASA organizations. Rack drawings and existing power requirement determinations, as 
well as providing circuit diagrams including power distribution units and breaker panels, were all required to be 
delivered to the construction companies. Diligence was crucial in maintaining and protecting POIC hardware assets 
in the ever-changing construction environment. The HOSC developed the HPOC, a smaller-scale operations control 
area, in an alternate building about a mile away from the HOSC. The criticality of the HPOC capability in preparing 
and implementing the required power outages hinged on the transition to and from the HPOC. A critical point for 
the facility was transitioning to the new mechanical and electrical systems. The team realized and identified the risks 
involved in transitioning from HPOC to HOSC. Miscommunications during the schedule development at several 
points, would have led to multiple outages across several racks of equipment impacting ISS operations. The team 
quickly coordinated among all stakeholders to mitigate what could have been a major issue and prevented any 
outages. Efficiency in the testing and validation of the new infrastructure with the power source change resulted in a 
24-hour delay in transition back to HOSC from HPOC. With several power outages during the PCA-1 upgrades, the 
HPOC was an optimum solution as an alternative location for the flight control team and support teams to continue 
uninterrupted operations. 


IV. Room Design 

Once the decision was made to turn the room to face north, different console layouts were considered. One 
option was an arc design: 3 full-length rows of consoles on a slight arc, plus one shorter arc in the back, Figure 6, 
left. This concept left too little room between the third row and the columns, making areas unusable. Another option 
was essentially the same, but without the arc, simply 4 straight rows of consoles, Figure 6, right. The flight control 
team requested a walkway down the center of the room, so that console operators could walk to different rows 
without walking all the way around the room. There was no need to break the first row since no one would need to 
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go to further than that to have a conversation. The center aisle also serves as a good collaboration area when several 
flight controllers need to have an over-the-wall conversation, that is, a conversation not on the voice loops. 



^ ^ 



r — i 


Figure 6. Initial PCA-1 layout concepts. 

A. Consoles 

The consoles chosen leveraged JSC’s investment on console design, using a vendor specializing in operator 
consoles and ergonomics. However, PCA-1 needed longer length consoles than JSC used due to the hardware 
footprint of ISS payload flight controller positions. The consoles are Evans * I 2 * * * * * * 9 , full-depth Strategy™ with rear 12” 
high slat wall. 

The slat wall can be used for mounting monitor arms, optional lighting, telephones, or other accessories. In PCA- 
1, monitors, communications units (Internet Voice Distribution System, IVoDS), and position name signs are 
mounted to the slat wall. Depending on the monitor configuration, some consoles also have their telephones 
mounted on the slat wall. The back of the slat wall is covered with acoustic panels to help with ambient noise in the 
room. 

A physical inventory of hardware was taken for both desktop and under-desk equipment, as shown in Table 3. 
The DMCs in PCA-1 have the biggest hardware and the Payload Rack Officers (PROs) have the most hardware, so 
the consoles were designed for those positions. 

back, left to right). 


Desktop 

A - Standard LCD (16” W x 13” H) 

B - IVoDS LCD touchscreen (12” x 9”) 
C - IVoDS processor (7.5” x 10.5”) 

D- Video LCD (20” x 13.5”) 

E - Video LCD (31.5” x 20.5”) 

F - Widescreen LCD (20” x 14”) 

G- Large LCD (27” x 18”) 

H- Large LCD (31”x20”) 

I - Phone 

J - Keyboards and mice 


Under Desk 

K - CPU Tower 

The old PCA-1 console layouts were studied with each console operator position team. New layouts were 

developed based on the new design of the room. For example, the Payload Communicators (PAYCOMs) decided to 
unstack their monitors so they have a better view of the video wall. The DMCs, with stacked 30” monitors, shifted 

their communication set placement to create a gap for face-to-face communication with the Payload Operations 

Director (POD), who sits in front of them, as shown in Figure 7. 


Table 3. Quantities of equipment for each console (front to 


Operator 

Quantities of Equipment 

A 

B 

C 

D 

E 

F 

G 

H 

I 

J 

K 

LIS Rep 

4 

1 

1 

1 





1 

2 

2 

PRO-1 

14 

1 

1 

1 





1 

4 

4 

PRO-2 

10 

1 

1 

1 





1 

3 

3 

PRO-3 

10 

1 

1 

1 





1 

3 

3 

Stowage 

3 

1 

1 

1 





1 

1 

2 

TCO 

2 

1 

1 

1 


2 



1 

2 

2 

OC 

7 

1 

1 

1 





1 

3 

3 

Ops Lead 

2 

1 

1 

1 





1 

1 

1 

PAYCOM-2 

4 

1 

1 

1 





1 

1 

3 

PAYCOM-1 

5 

2 

2 

1 





1 

2 

4 

POD 

3 

1 

1 

1 




1 

1 

1 

2 

APOD 

5 

1 

1 

1 


1 



1 

1 

2 

DMC-2 


1 

1 

1 

1 


3 


1 

2 

2 

DMC-1 

1 

1 

1 

3 



6 


1 

2 

4 
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Figure 7. Equipment Configuration of PRO and DMC consoles in PCA-1. 


The tops of the PCA-1 consoles have continuous Plexiglas™. This choice saved the cost of embeddeding the 
Plexiglas™ flush with the rest of the console surface and provides more room for reference material storage 
underneath the “glass.” The Plexiglas™ is split in three sections across each console (left, middle, right) to reduce 
the weight, so operators can lift it up. Plexiglas™ also covers the triangular millwork connectors between each 
adjacent console. 

Each console is equipped with a power and data outlet at each end of each console, flush mounted into the 
bottom of the slat wall system. Each outlet provides 2 power and 1 ethemet connections. Flight controllers generally 
bring their work laptops to a console shift as a backup resource and to get other work done as time allows. Having 
the power and data connections on top of the console means that no one has to crawl around on the floor under the 
desk to plug in their device. 

Flight controllers typically keep a vast amount of reference material at their fingertips for use in both emergency 
situations and nominal operations. Accomodations for storage, either in the consoles themselves, or in nearby 
bookcases was required. The Strategy™ consoles have bays underneath the desktop, which open from both front 
and back, for slide-out processor trays, storage, or rack mounted equipment. The PCA-1 consoles have 5 bays each. 
One bay was configured for operator storage and the other 4 bays were configured with slide-out processor trays. 



Figure 8. DMC-2 and DMC-1 console drawings. 
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Figure 9. PRO-2 and PRO-3 console drawings. 


B. Amenities 

In addition to the 
consoles, several 
amenities were also 
purchased to round 
out the room design 
with matching 

furniture. There are 
six bookcases of 2- 
shelf height designed 
to fit 3 -ring binders. 

There are six similar 
bookcases with doors. 

The open bookcases 
are positioned along 
the wall underneath 
the viewing room 
windows since tour 
groups or the 
television cameras 
cannot see them. 

Three of the closed 
bookcases are along 
the mural wall and 
three are on the back 
wall. There are two 
extended depth 

cabinets designed for 
printers. The nook on 
the backside between 

each pair of adjacent consoles has a contoured bookcase, also designed for 3 -ring binders, to provide additional 
storage to the person who sits in front of the adjacent consoles. Adjacent consoles are joined with a pie-shaped 
storage wedge, providing a continuous look down each row. 

Before purchase, the flight control team tested several ergonomic chairs. PCA-1 has 2 chair designs so that each 
console operator can make a selection for proper fit and function. 



Figure 10. PCA-1 room layout, with bookcases pictured. 


11 

American Institute of Aeronautics and Astronautics 



Each console has a lighted position call 
sign fabricated from engraved glass that is 
mouted to the slat wall. Each sign has a 
remote that allows the operator to change the 
color of the sign. PCA-1 flight controllers 
use them as another means of 
communication. For example, the POD will 
ask the team to change to a specific color 
when they are ready for a certain operation or 
have finished a certain task. 

Figure 11. Lighted position call sign. 




C. Viewing Room 

The Viewing Room 
also got an upgrade 
during the PCA-1 re- 
engineering effort. The 
viewing room contains a 
light box on two walls of 
the room, showcasing 
MSFC’s contribution to 


experiments operated by 
the POIC. MSFC 


engineers designed the 
environmental control 


and life support systems, 
water recovery and 
filtration system, and 
EXPRESS Racks, the 
refrigerator-sized 
facilities that house most 


experiments. The light Figure 12. PCA-1 from the Viewing Room, 
box also highlights the 

current ISS crew and some of the science they will perform during their expedition. The viewing room has a 
television and DVD player that gives a guided tour through PCA-1 and describes the jobs of each cadre position in 
the room. All of the light box photos were replaced and updated during PCA-1 refurbishment. The PCA-1 guided 
tour video was completely re-filmed in the new PC A- 1 in one day before the cadre moved in. 


D. Room Layout 

Once the room orientation, consoles, and basic layout were determined, a great deal of thought went into 
deciding where each position would sit in the room. The huge monitors of the DMCs would be an obstacle to any 
flight controller’s view of the video wall, so they needed to be either in the first or last row. Likewise the PROs have 
10-14 normal sized monitors stacked, creating a wall. The old PCA-1 had 2 PRO conosles and 2 DMC consoles and 
we were looking to create a third PRO console for future capability increase in the new design. Splitting the PROs 
between different rows did not make sense, so the 2 DMC consoles went on the back row and the 3 PRO consoles 
went to the front row. 

In the old PCA-1, POD sat in front of DMC and this arrangement was valuable for quick over the wall requests 
for video, an onboard resource that DMC controls. Additionally as leader of the flight control team, the POD wants 
a good view of the whole room, so sitting towards the back provides that perspective. The cosnsole to the right of 
POD is used for shift handover. POD participates in the Flight Director’s briefing at the beginning of each shift, so 
the Alternate POD (APOD) console provides a space for that activity. This console is also used when POIC staffs a 
subject matter expert POD for a complex operation. 

The POD has a lot of non-verbal communication with the PAYCOM, so having them sit on the same row 
enables them to communicate via eye contact or hand gestures. The Operations Controller (OC) is the POD’s 
backup in the room and also the real-time safety officer, so they needed to be close to POD. This put them in front of 
POD, in the second row. The console next to OC was intentionally assigned to positions that normally do not staff 
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PCA-1, Safety and Operations Lead. These positions staff very rarely, but when they do, it is for a very good reason, 
like a safety incident or complex operations. The intent was that this normally empty console can also be used by the 
OC team when they want to staff a second OC for a complex task. The Timeline Change Officers (TCOs) wanted to 
be closer to POD than they were in the old PC A- 1 layout, so they were placed across the isle in the second row, next 
to OC. Stowage typically interacts most with PAYCOM and TCO, so they went next to TCO on the second row. 

The 3 PROs were arranged on the front row in the right three spots. Currently POIC staffs 2 PROs on most shifts 
and only brings in a third PRO when there are complex ops and an expert is required. Since PRO-3 is not always 
staffed, we decided to put them furthest from POD (front row, all the way to the right). So if POD wants to have an 
over the wall conversation with the PROs, most of the time they just walk up the center aisle and need not transverse 
the row. The Lead Increment Scientist Representative (LIS Rep) attends a lot of meetings with the science 
community, so the spot remaining works well for them as they are on an outside aisle near the door. Figure 13 shows 
the final arrangement of PC A- 1 . 



Figure 13. Layout of PCA-1 console positions. 


V. Information Technology 


A. Video Wall 

With more than 200 experiments 
being conducted on the ISS, the need 
for information sharing among 
ground controllers led to the need for 
a video wall. RGB Spectrum 1 was 
selected as the vendor for the video 
wall. Composed of a 2x12 
configuration of monitors using a 
display processor to route and display 
up to 32 input signals, ground 
controllers are now enabled to share 
information in real-time, such as live 
video feeds, diagrams, photographs, 



Whiteboard 

Fault 

Downlink Video 

Other Video & 

Ground 

Summary 

Channels 

PCA1 PC Displays 

Track 



TV dimensions 


Figure 14. PCA-1 Video Wall, including dimensions and video 
sources. 
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results of experiments, and information on power usage. Each ground controller uses a console computer to execute 
applications related to their areas of expertise, and signals from each computer are routed to the processor. Other 
inputs to the wall display include feeds from on-board video cameras, the ISS ground track, which includes the 
ISS’s projected path along a mercadian projection of the earth with satellite and ground station information, news 
broadcasts, weather information, and the NASA Television channel. The video wall enables fast source switching 
and collaborative management of large amounts of information supporting real-time decision making, optimizing 
the support for scientific discovery. 

The design and 
procurements for the 

Video Wall in PCA-1 were 
executed within an 

extremely compressed 
schedule environment. The 
contractor excelled in 
pulling together the 

requirements and pushing 
forward with a solution in 
a very short period of time. 

Their plan to use corporate 
resources aided in 

exceeding the schedule for 
the overall control room 

upgrade project. The 
expertise that was brought 
in aided in a quick and 
professional installation 
that helped lower costs and 

support schedule objectives. The end product was so well received that the vendor would like to use the PCA-1 
installation as a showcase for their marketing materials. 

Once the video wall and controllers were installed, the vendor had to train the flight controllers to operate the 
processor software. This was quickly accomplished using the train-the-trainer approach. One flight controller from 
each of the teams who have a video wall controller at their console was trained to operate it. Then those flight 
controllers in turn taught their teams. It was a very effective plan given the short amount of time available before the 
cadre moved back to PCA-1 and was successful because the touch screen controllers are intuitive to operate. 

B. Network Infrastructure System 

A design conducive to the ease of PCA-1 control room technician’s ability to provide technical support to 
services and maintenance led to upgrading the PCA-1 utilization of an Ethernet hub design. Research led to this 
implementation for a new network infrastructure system. The new infrastructure uses Ethernet trunk connectivity in 
lieu of individual cables. One Ethernet trunk provides six Ethernet connections. By using one trunk at each console, 
the PCA-1 connections were reduced from 84 individual cables to 14 trunks. This technology provided enormous 
ease of installation as well as far more flexibility in how the room was outfitted along with providing flexibility for 
future growth. A color-coding scheme used on all the trunks further simplified the installation and decreased the 
time required for trouble shooting any problems that might occur. This technology decreased the Ethernet cabling by 
approximately 85%, which proved to be a very cost effective solution. 

C. ISS Live 

As the focus of the ISS shifted from assembly to research, the ISS Program decided that POIC should be 
regularly featured in the daily ISS Update program shown on NASA TV. The start date for the new feature was May 
1, during PCA-1 refurbishment. Temporary cameras were installed in PCA-2 while the cadre operated from there. 
Three high definition, remotely operable cameras were installed in PCA-1 for use when the cadre transitioned back 
to the new PCA-1. To support this new capablility, equipment had to be installed and routed within the HOSC and 
from the HOSC to the MSFC Television facilities. 



Figure 15. Wide-angle photograph of PCA-1, with video wall. 
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VI. Lessons Learned and Future Work 


Primary Lessons Learned from this project was the essential need for management buy-in from conception. Also, 
the Sprint methodology and face-to-face team meetings removed roadblocks that allowed the team to accelerate the 
schedule. The schedule was a powerful tool used as a task list to keep the team focused on upcoming activities and 
gave the entire team insight into actions required to meet the next milestone. Building on existing control room 
technologies and layouts accelerated the schedule. The use of visualization tools gave the team confidence in 
selections of consoles and IT equipment prior to ordering. Engineering drawings from the vendors as well as having 
equipment assembled as much as possible prior to setting up the room expedited the final assembly and framework 
of the room. Transition checklists for moves and room build-ups led to fewer unexpected issues, which ultimately 
sped up the transition process. 

A. Ergonomics 

The ergonomic design of a control center is essential is the efficiency and effectiveness of the heart of the room, 
the Operators. Meetings were held with the MSFC Ergonomics organization to educate the team on what to look for 
and which items to consider. Chair selections were made per console operator for proper fit and function and were 
achieved through the vendor supplying an array of chairs for which each operator was fitted. The console vendor 
was selected based on their specialty in operator console designs and ergonomics, considering lines of sight, 
hardware equipment, and room configurations. The ability to receive a console earlier would have reduced the 
configuration time and allowed us to understand the human factor element to completeness. 

B. Acoustics 

The original PCA-1 was evaluated for acoustics, which resulted in a white noise acoustical system being 
installed in the room and sound soak provided as part of the consoles. The white noise system remained in the room 
and the new replacement consoles were ordered with sound soak. As we moved the controllers back into PCA-1 and 
began operations, it became clear that the acoustics had changed with the re-orientation, new console material, video 
wall, and new paneled wall amenities. An acoustics assessment and study during requirements gathering might have 
allowed us to understand how the noise level in the room would change with the re-orientation, new consoles, video 
wall and amenities. 

C. Training 

The aggressive schedule did not allow proper time to train either the Integrated Support Team (1ST) or the cadre 
on the video wall capabilities, control functions, and troubleshooting. 

D. Desktop Virtualization 

As with the video wall and the need for information sharing, the need for processor power to control, command, 
and configure onboard payload resources has also increased. As referred earlier with the physical consoles in Table 
3, two flight control positions have utilized as many as four personal computers to drive multiple displays for a 
variety of onboard functions. The PRO configures and monitors various ISS rack services to sub-rack payloads as 
well monitoring the health and operations of these racks. The PRO displays a variety of telemetry information of 
these systems as well as command responses to these resources. The PRO also controls when payload developers 
can do direct commanding to their own payload. The DMC also requires many personal computers to drive displays 
like the PRO. The DMC controls the configuration of network resources, data and video configuration and routing, 
as well as onboard data system troubleshooting activities. The DMC also manages the Ku-band uplink and downlink 
utilization, a precious resource, for a multitude of ISS users, including both systems and payloads. 

It always seems that with more tools, more automation, and more control of program assets, the need for 
additional computers at the PRO and DMC consoles has outpaced the increase in the evolution of central processing 
unit (CPU) power. These controllers were requiring more displays and complaining that their computer systems 
were performing slower over time. Another advantage of the video wall was to alleviate the number of monitors at 
each console station. Instead of each console position showing the same display, it could be shared on the video 
wall, reducing the number of monitors by eliminating this duplication. Another new technology could also solve the 
ever-increasing need for more CPU power and eliminate the problem of slow systems. Desktop virtualization, 
particularly at the PRO and DMC consoles, could provide the POIC with flexibility to always have the processor 
power available when needed. The HOSC is currently investigating this new technology in a control center 
environment by testing a Virtual Desktop Infrastructure (VDI). The user desktop environment, including 
applications and services, are located on servers in the central computer center. While the HOSC does utilize virtual 
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server technology, as do most control centers, virtual desktops will aid the PRO or DMC during those peak times 
when they require more resources. The number of CPUs at their console will no longer limit them. 

Under the old information technology models, the HOSC provides the number of computers required for periods 
of maximum utilization. Using the VDI model, the PRO or DMC will always have, on demand, the processor power 
needed for scheduled or unscheduled activities. Freeing up much needed console space, the HOSC is testing zero 
clients at the console. The zero client is a much smaller box, typically the size of the human hand. It provides the 
drivers for locally attached devices, such as keyboards, card readers, mouse, and monitors. It provides the network 
connectivity to the servers that house the desktop environment. In the event of a hardware failure, these devices are 
quickly removed and replaced. The console operator is back in operation much sooner than delivering, installing, 
and configuring a new personal computer. While the video wall reduces the number of monitors needed for a 
console position, the flight control team still requires multiple display options. Extensive testing is currently 
underway to evaluate various zero client product offerings. Two main factors in the evaluation are the display 
drivers and the number of monitors that a zero client can drive. Image resolution quality requirement is increasing as 
more displays are evolving from just text and colors to more elaborate models and images with embedded telemetry 
and alarms. The flight controllers are awash in information being displayed. Hopefully, the VDI in conjunction with 
the video wall will optimize how this information is displayed for more rapid and more intelligent team responses to 
ISS events. 

E. Other Control Rooms in the HOSC 

The HOSC would like to re-engineer the Data Operations Control Room (DOCR) following the methods used in 
the PCA-1 re-engineering. The DOCR provides a critical support function to each and every control room in the 
HOSC. For example, the DOCR staff are responsible for ensuring that all of the POIC cadre’s systems are 
operational and that the ISS scientific data gets from the HOSC to scientists all over the world. The HOSC will be 
the future home of the SLS Engineering Support Center and other small satellite control rooms. The HOSC team 
will take the lessons learned from the re-engineering of PCA-1 and apply it to these future upgrades. 

VII. Conclusion 

In conclusion, the PCA-1 re-engineering effort would not have been successful and could not have been 
performed so quickly without an incredible amount of planning and teamwork. 

On June 19, 2013, Mike Suffredini, ISS Program Manager, paid a visit to Huntsville’s MSFC for the unveiling 
of the new control room. As Mr. Suffredini spoke with news stations, newspapers, and the Public Affairs Office, he 
was joined by the MSFC Deputy Center Director, Engineering Director, ISS Payloads Manager, and the MSFC 
Mission Operations Director. 

Excerpt from the NASA MSFC News Release 3 - June 19, 2013: NASA unveiled today an upgraded Payload 
Operations Integration Center at the Marshall Space Flight Center in Huntsville, Ala. The operation center’s new 
capabilities enhance collaboration and enable the ground team to efficiently help the International Space Station 
crew and researchers around the world perform cutting-edge science in the unique space environment. 

“Conducting cutting-edge research that benefits space exploration as well as life on Earth is a top priority for 
the space station,” said Michael Suffredini, manager of NASA’s International Space Station Program. “With this 
amazing in-space laboratory now fully functional, the crews are able to dedicate more time each week to scientific 
research and the payload operations team at Marshall has had a major role in making that happen.” 

"Over the course of the last 12 years, our team has learned much about how they can collaborate to maximize 
science return,” said Jay Onken, manager of the Mission Operations Laboratory at Marshall. "They used this 
knowledge to redesign the control room to have the most modern technical equipment to support the most amazing 
international engineering and scientific endeavor of the century.” 

"We recently achieved a major milestone exceeding the goal of completing an average of 37 hours of crew- 
tended science per week across a six month period," said Carmen Price, leader of the payload operations integration 
function at Marshall. "Our team even helped the crew achieve a record 72 hours of crew-tended science experiments 
— the most hours of science ever conducted by a space station crew in a single week. While the crew is sleeping, we 
are here conducting experiments remotely from Earth, ensuring numerous automated experiments have the power 
and data recording and transmission needed to operate successfully." 
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